HANDBOOK OF FUNCTIONAL NEUROIMAGING OF COGNmON

Semantic Representations Thus far, our discussion of the representation of semantic memory has been confined to physical properties of concrete objects. When Allport articulated his theory of the representation of meanings, he limited himself, "for simplicity," to the domain of object concepts. Of course, any complete theory of semantic memory has a bit more work to do: there are abstract concepts (e.g., peace), abstract features of object concepts (e.g., "alive" as a feature of a plant), and abstract relations between concepts (e.g., the ways in which a "bat" is more similar to a "bear" than to a "bird"). There is ample neuropsychological evidence for a dissociation between the representation of abstract and concrete concepts (e.g., Breedin et al., 1994) that may reflect any number of qualitative differences in their acquisition and representational format. Neuroimaging comparisons of abstract and concrete words have identified an inconsistent array of regions associated with abstract concepts in the left superior temporal gyrus (Wise et al., 2000), right anterior temporal pole (Kiehl et al., 1999), and left posterior middle temporal gyrus (M. Grossman et al., 2002). Noppeney and Price (2004) compared fMRI activation while subjects made similarity judgments about triads of visual words, sound words, action words, and abstract words that were matched for difficulty. Relative to the three other conditions, retrieval of abstract concepts activated the left inferior frontal gyrus, middle temporal gyrus, superior temporal sulcus, and anterior temporal pole. The authors suggest that these differences reflect activation of areas involved in sentence comprehension, although this is clearly an area in need of more investigation. There is, to date, even less work addressing abstract features. One particular problem with using functional neuroimaging to compare abstract and co,?crete features is that abstract semantic decisions typically take longer to resolve. The left inferior PFC has been implicated in extended or controlled semantic processing on the basis of studies that might confound effort with abstractness (Roskies et al., 2001); for example, PFC activity increased when subjects decided that "candle" and "halo" were similar, compared to deciding that "candle" and "flame" were similar 165 Functional Neuroimaging of Semantic Memory (Wagner et aI., 2001), arguably a comparison that confounds effort with abstraction. To unconfound these processes, Goldberg and colleagues (2004) compared the effects of increasin& semantic abstractness and increasing difficulty on activity in PFC while subjects verified perceptual or abstract facts about animals. fMRI activation in left PFC (BA 47) was specifically associated with an increased reliance on abstract properties but not increased semantic difficulty. This finding is consistent with recent evidence showing that ne_urons in a primate analogue of this region represent abstract rules (Wallis et aI., 2001). Many cognitive models of semantic memory have described hierarchical networks that reflect abstract relations between concepts (e.g., a tree is a plant and a plant is a living thing). This description is quite different from the distributed representation we have been describing thus far. However, Rogers and colleagues have articulated a fonnal model of semantic memory that includes units which integrate infonnation across all of the attribute domains (including verbal descriptions and object names; McClelland & Rogers, 2003). As a consequence, "abstract semantic representations emerge as a product of statistical learning mechanisms in a region of cortex suited to performing cross-modal mappings by virtue of its many interconnections with different perceptual-motor areas" (Rogers et aI., 2004, 206). The interaction between content-bearing perceptual representations and verbal labels produces a similarity space that is not captured in any single attribute domain; rather, they argue, it reflects abstract similarity (d. Caramazza et aI., 1990; A. R. .,,-, Damasio, 1989; Plaut, 2002; Tyler et aI., 2000). as a candidate for these abstract representations is the temporal pgle, based both on the anatomical connectivity of this region and the degener-atien:lOf this region in semantic dementia. The notion that interactions between perceptual and verbal representations lead to the emergence of new, abstract representations may be relevant for a puzzle that has emerged in neuroimaging tests of Allport's (1985) sensorimotor model of semantic memory: that there is a consistent trend for retrieval of a given physical attribute to be associated withactivatlon 6f'coiifcal et aI., 1995; Thomps'on-Schill, 2003). This pattern, which has been interpreted as coactivation of the "same areas" involved in sensorimotor processing, as Allport hypothesized, could alternatively be used as grounds to reject the Allport model. What does this anterior shift reflect? We believe the answer may lie in the ideas developed by Rogers and colleagues (2004). The process of abstracting away from modality-specific representations may occur gradually across a number of cortical regions (perhaps converging on the temporal pole). As a result, a gradient of abstraction may emerge in the representations throughout a given region of cortex (e.g., the ventral extrastriate visual pathway), and the anterior shift may reflect activation of a more abstract 166 s. L. Thompson-Schill, L P. Kan, and R. T. Oliver representation (Kosslyn & Thompson, 2000). The tasks that have been used to study conceptual retrieval of visual attributes have not consistently required the subject to retrieve perceptual information. For example, in order to recall that a banana is "yellow," activation of color representations that are more abstract than those necessary for perception could suffice. The conceptual similarity space in more anterior regions may depart a bit from the similarity space in the environment, perhaps moving in the direction of abstract relations. More work is needed to uncover the nature of the representations-and how the similarity space may gradually change across different cortical regions. Categories of Semantic Memory-Redox Thus far, we have presented two potentially orthogonal views about the organization of semantic memory. We initially considered the hypothesis that representations of specific categories of semantic knowledge are instantiated in spatially distinct neural regions. As we saw, there is ample support for this hypothesis from the neuropsychologicalliterature, but only partial support from neuroimaging studies. Then, we reviewed neuroimaging studies that support models of distributed representations of semantic memory, where different attribute domains of object knowledge are represented in distinct sensorimotor systems. As the reader having even a passing familiarity with these literatures will know, these two hypotheses about the organization of semantic memory are intertwined, by virtue of the fact that the taxonomic category of an object and its associated attribute domains are not at all orthogonal. The confound between these two putative organizing principles has made it challenging to uncover the neural architecture of semantic memory. Warrington and McCarthy (1983) first called attention to implications of this relation for the interpretation of category-specific deficits: whereas sensory attributes are important for discriminating between members of the category of living things, functional attributes are more important for discriminating between members of the category of nonliving things. Thus, category-specific deficits could result from the degradation of an attribute domain of semantic memory. Since the mid-1980s, this sensory-functional theory has persisted in a variety of accounts of categoryspecific deficits, all of which hold that semantic knowledge is stored in sensorimotor channels, and that the relative importance of information contained in these channels varies across items in different categories (e.g., Farah & McClelland, 1991; Martin et aI., 2000; Saffran & Schwartz, 1994; Simmons & Barsalou, 2003). Accordingly, one explanation of category-specific activation in neuroimaging studies is that these differences reflect the differential weighting of visual and functional knowledge across categories (e.g., Martin et aI., 1996). In order to test this account, Patterson and colleagues have reported two PET studies that have uncon167 FlUlctional Neuroimaging of Semantic Memory founded object category and attribute domain. In the first, subjects made similarity judgments about living or nonliving things on the basis of either visual or nonvisual information. With this fully crossed design, the authors compared the magnitude of category-specific effects and attribute-specific effects directly, and concluded that the latter were more prominent neurally (Mummery et al., 1998). In a second study, subjects generated visual or nonvisual features in response to an object name (Lee, Graham, et al., 2002). Although they found no category-specific effects, they did find an effect of attribute type: visual retrieval activated left posterior inferior temporal cortex, and nonvisual retrieval activated left middle temporal cortex and right fusiform cortex. The relationship between visual processing demands and object category was elegantly demonstrated in a PET study by Rogers and colleagues (2005). Subjects categorized photographs of animals and vehicles at one of three levels ofspecificity (e.g., animal, bird, or robin; vehicle, boat, or ferry). Posterior fusiform activation was greater for animals than for vehicles only when subjects were categorizing pictures at an intermediate level (e.g., bird). The authors argued that the fusiform gyrus responds to the discrimination of items with similar visual representations, and that at the intermediate level of description only, animals have more overlapping visual properties than do vehicles. In addition, the modulation of the category effect by task demands provides a plausible explanation for the inconsistent pattern of category-specific effects described earlier. The sensory-functional theory has been debated and refined as new observations have challenged the ability of this theory to parsimoniously account for the relevant neuropsychological data. Caramazza and colleagues have frequently called attention to some of the more problematic findings for the sensory-functional theory. An early objection was based on the observation that patients with living-things deficits have impairments across multiple attribute domains (Caramazza & Shelton, 1998). The sensory-functional theory, which presumes that living-things deficits result from loss of visual knowledge, would seem to predict normal nonvisual knowledge of living things. This objection was initially answered by Farah and McClelland (1991), who used a computational model to demonstrate the emergence of category-specific effects (across attributes) following damage to an interactive, attribute-specific systems. Key to the behavior of this model was the assumption that retrieval of a weakly represented attribute of a concept would depend on the activation of more strongly represented attributes, thus exhibiting a critical-mass effect. ThompsonSchill and colleagues (1999) sought physiological evidence for this assumption: for living things, retrieval of visual or nonvisual information should require activation of visual representations, because of the disproportionate weighting of visual information in the representations of living things. For nonliving things, no such 168 S. L. Thompson-Schill, I. P. Kan, and R. T. Oliver dependence on visual knowledge should occur. As predicted, areas involved in visual knowledge retrieval were active during judgments about visual and nonvisual attributes of living things but only during judgments about visual attributes of nonliving things. These results lend credence to claims that category-specific activations actually reflect attribute-specific representations. (For a different interpretation of these data, see Caramazza, 20(0). A second criticism of Caramazza and colleagues' has proven more difficult to answer: the sensory-functional theory would seem to predict that patients with a degradation of the visual attribute domain would have impaired visual knowledge of all concepts, not just of living things. However, at least some patients with a livingthings deficit have normal visual knowledge of nonliving things (Caramazza & Shelton, 1998). Here, we suggest a possible way to answer this objection, on the basis of some of the ideas that have emerged from the neuroimaging studies reviewed above. As we argued earlier, visual knowledge is most probably not a single attribute domain. Under this revised description of visual knowledge, in which visual knowledge itself is a distributed representation, a different set of predictions emerges: objects with multiple sources of knowledge about their appearance (e.g., vision, touch, actions) will be less susceptible to loss of any single source of visual knowledge (ct. Crutch & Warrington, 2003). We tend to have more sources of knowledge of the appearance of nonliving things, or at least of certain nonliving things. Thus, damage to ventral visual processing regions, which represent only one source of information, will not necessarily cause an impairment to other representations of appearance for these things. This idea was present in Allport's (1985) description of attribute domains (he used the example "cloud"), but it was not included in many sensory-functional theories that, in effect, collapsed across all types of visual knowledge. We argue here that the consideration of multiple sources of visual knowledge-and the way those sources vary across categories-may be crucial to our ability to explain category-specific phenomena. There are some provocative data that lend credence to this conjecture: Borgo and Shallice (2001) described a patient with a living-things deficit who was also unable to identify nonliving things without a solid form (e.g., liquids). They argued that the affected attribute domains were purely visual qualities, such as color and texture, which are unrelated to object use. However, his knowledge-including visual knowledge-of artifacts presumed to have strong form-action links (i.e., affordances) was preserved (ct. De Renzi & Lucchelli, 1994;Tyler & Moss, 1997). Wolk and colleagues (2005) more directly examined the role of affordances in a patient with an apparent living-things deficit. They noted that this patient was impaired at recognizing not only animals but also artifacts that minimally afforded a particular action. In contrast, for artifacts that strongly afforded an action (e.g., piano), the patient could identify a line drawing of the object. The patient's ability to recognize shape, in the 169 Functional Neuroimaging of Semantic Memory absence of a functional occipitotemporal representation of form, may have been mediated by action representations (for objects where the form affords the action). Subsequently, we demonstrated that this patient's knowledge of the color of objects was impaired (in contrast to his normal knowledge of both shape and size; Oliver et aI., 2004). In summary, the relationship between taxonomic categories and attribute domains, and the implications of that relationship for our understanding of phenomena such as category-specific deficits and activation patterns, is continuing to be informed by new neuroimaging studies of semantic memory. It is likely that at least some of the category-specific phenomena will be better understood as the result of processing within a distributed semantic system organized around a broad collection of sensorimotor attributes. However, refinements to the sensory-functional theory-perhaps beginning by abandoning the term "functional"-are clearly warranted by both the neuropsychological and the neuroimaging literature (figure 6.1). Finally, it is worth noting that evidence for attribute-specific representations does not necessarily refute the hypothesis that there are category-specific representations (and vice versa); it is possible that the organization of semantic representations has more than one governing principle. Several investigators have proposed the emergence of category representations as an intermediary between sensorimotor knowledge and language (e.g., Coltheart et aI., 1998). The relationship of semantic memory to language, and the extent to which category-specific representations exist in either or both, should be the subject of future research. tactile manipulation .' / )spatiomotor ..-e I .1/ elements .. ... -"-. (' )./ .-..... , .....? '...' e >:. " taste u /'.....) . Visual elements elements cf color Figure 6.1 A revised version of Allport's (1985) influential model of distributed sensorimotor semantic represen· tations, incorporating attribute domains that have been the subject of recent neuroimaging investigations. 170 S. L. Thompson-Schill, I. P. Kan, and R. T. Oliver Semantic Memory Retrieval In the preceding section, we argued that semantic memory comprises, at least in part, a distributed set of representations that are tied to sensorimotor channels. Now we turn to the question of how these distributed representations are retrieved when one accesses semantic memory. In particular, we focus on two questions: First, we consider the relationship between the input modality and the process of semantic retrieval. Second, we discuss evidence that regions of PFC, in certain contexts, function to bias activity across distributed representations in semantic memory. Accessing Semantic Memory from Words and Pictures In neuropsychological investigations of semantic memory impairments, striking dissociations have been noted between visual and verbal input modalities. For example, patients with optic aphasia are unable to name visually presented objects, despite relatively spared perception of stimulus surface structure (Beauvois, 1982; Riddoch & Humphreys, 1987), and other patients perform significantly better with pictures than with words (e.g., Bub et aI., 1988; Lambon Ralph & Howard, 2000; Saffran et aI., 2003). These dissociations have led researchers to examine whether information from different input modalities may have differential access to different content within the conceptual system (see also Paivio, 1971; Shallice, 1988). Many of the neuroimaging studies examining modality differences have reported regions of common activation and regions of modality-specific activations. For example, Vandenberghe and colleagues (1996) reported that semantic judgments of both words and pictures activated common regions in inferior temporal and frontal cortex, but also that a few areas were uniquely activated only by pictures (left posterior inferior temporal sulcus) or only by words (left anterior middle temporal gyrus and left inferior frontal sulcus). Similarly, Postler and colleagues (2003) observed common areas of activation across verbal and visual modalities (left inferior frontal gyrus and middle temporal gyrus) along with modality-specific areas (see also Bright et aI., 2004 for a meta-analysis and report of similar findings). Across various neuroimaging studies, a few regions have emerged as candidate regions for an amodal semantic system: left inferior frontal gyrus, middle temporal areas, and ventral temporal lobe, centered on the fusiform gyrus (see also Bookheimer et aI., 1995; Moore & Price, 1999; Perani et aI., 1999). Since activity in these common regions is invariant to input modality, these data seem to provide support for a unitary semantic system, such as that described by Caramazza and colleagues (1990). However, one must be cautious in interpreting "common activation" as "common representations." Given the limited spatial resolution of fMRI and PET, it is difficult to determine whether commonly activated brain regions indicate involvement of the same network of neurons or involvement of different networks of neurons exist171 Functional Neuroimaging of Semantic Memory ing in the same regions. One possible way to sidestep this limitation is discussed in "Issues" (below). Neuroimaging evidence for input modality-specific activations proves equally problematic to interpret. One possibility is that these patterns reflect the existence of separate visual and verbal semantic systems (cf. Warrington & Shallice, 1984). Under this account, there is a redundant representation of semantic information in two different formats. A second possibility is that modality-specific activation patterns reflect differences in presemantic processing (Bright et al., 2004). This account might explain why the locations of putative modality-specific regions have been inconsistent across studies. We favor a third explanation: different input modalities may be preferentially associated with (or have preferential access to) different attribute domains in the distributed semantic system. By this logic, modality-specific effects do not reveal differences in the format of the representations accessed by different modalities, nor do they indicate redundant representations of the same semantic information. Rather, under this account, modality-specific effects reflect relations across attribute domains. For example, consider the relationship between form and manipulation knowledge (described in the Form section): Pictorial stimuli, which contain form information, may have preferential access to manipulation knowledge compared with word stimuli. Consistent with this claim, Chainay and Humphreys (2002) reported that normal subjects were faster at making action decision (e.g., pour or twist?) about picture stimuli than word stimuli. Also, using a free association task, Saffran et al. (2003) observed that subjects generated more action words (i.e., verbs) in response to pictures of objects than to written names of objects. Thus, activation patterns that appear specific to pictures could instead be pointing to areas specialized to represent action information. These three accounts of modality-specific activations have yet to be distinguished within the neuroimaging